spec : add DFlash2 support (local convolution + candidate selector) - #27342
Conversation
|
Hi @SubSir, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
I've revised the PR description. It's fully hand-written now. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…and DFlash v1 Adds docs/dflash2-strix.md with the four-arm llama-benchy comparison on Qwen3.8-27B UD-Q4_K_XL, and vendors the unedited tool output under benchmarks/results/dflash2-20260819/. Headline, decode t/s at depth 0 / 8k / 32k: base (no spec decode) 11.81 11.44 10.54 DFlash v1 (n=5) 21.09 12.87 10.75 DFlash2 (n=4) 26.39 21.58 21.11 DFlash2 (n=7) 25.18 21.46 16.32 DFlash2 holds about 2x over base at every depth measured, while DFlash v1 decays to 1.02x by 32k, i.e. by then it is worth nothing over plain decode. Draft width matters at depth: at 32k, width 4 beats width 7 by 29 percent, and the two tie at shallow depth. DFlash2 engine support is upstream ggml-org/llama.cpp#27342 by Jian Chen and is not our work; the page says so. What is ours is the gfx1151/Vulkan validation and the draft-width tuning. Drafter weights are inco's published Apache-2.0 GGUFs. Measured with llama-benchy 0.4.0, which draws prompts from a Gutenberg book so acceptance reflects real prose. Content matters more than expected here: a code-corpus prompt set decays about 47 percent from shallow to 32k where this corpus decays about 20 percent, so the page carries that caveat rather than presenting one number as definitive. --runs 2, so the spec arms carry +/- 1.0 to 2.5 and only the 32k width gap is comfortably outside the noise. Prefill was taken at the server default -ub 512; -ub 256 is the measured dense optimum on these weights, so that column is a few percent below the ceiling. Noted in the page, rerun deferred. Assisted-by: Claude Opus 5
This comment has been minimized.
This comment has been minimized.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@Shamish There is already a fix for the reported problem at z-lab#1 As for the Performance on a single RTX3090, it seems to be not significantly better than MTP: MTP: DFlash2 (using maxn=4 because 7 was slower): |
This comment has been minimized.
This comment has been minimized.
|
Thanks for the PR. It would be great if you could run the benchmark on SpeedBench using the corresponding checkpoints and report the performance numbers. @SubSir |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
CUDA graphs ruled out too — on the slow build (2474373), So the elimination list is now: top-k kernel ❌, draft sampling ❌, p_min ❌, CUDA graphs ❌ — and notably the regression is NVFP4-specific on my side too (DFlash2 decode on a Qwen3.8-27B IQ3 target is unchanged between the two heads). That pattern is consistent with your second hypothesis: the NVFP4 post-scale fusion path. Whatever it is, it lands in the first 4 commits (1deefcc..2474373). |
I took nsys to qwen3.8 + dflash2 on the B4500. Per amdahl's law, we would expect up to ~1.3% E2E speed-up if me make top-k instantaneous (6 / 450 ms). While we are 2x faster at the kernel-level, this translates to an expected perf gain of 0.5% (which drowns in the other noise). What we should be looking at instead to accelerate DFLASH2 in llama.cpp (viewed through the eyes of a CUDA engineer):
Best |
|
Completing the retraction with the head commit — greedy bench (temp=0, warmup discarded, 6 runs): 2f3923b = 186/188/185/189/189/186, avg 187 — identical to the 5ecbe1a baseline (188 avg). Full greedy picture:
No regression anywhere in the window — the PR is performance-clean from my side. (Also noted: first-generation-after-load runs ~10% slow — always warm up before measuring.) |
Evaluate the algorithmic DFlash2 selector boundary without importing an untrained convolution layer. Prior-art: ggml-org/llama.cpp#27210; ggml-org/llama.cpp#27342 Signed-off-by: Codex <codex@openai.com> (fak model)
FYI for anyone benchmarking this on VulkanDFlash 2 verify graph triggers a pre-existing bug #27805 in ggml_vk_graph_optimize (ggml-vulkan.cpp, not touched by this PR). The optimizer does not see dependencies between different views of one tensor and can reorder across them, so on Vulkan the main model accepts draft tokens it did not choose: wrong output at temperature 0. CUDA is not affected. Fix in #27812. Until a fix is implemented upstream, Vulkan acceptance and speed numbers for this PR are invalid. GGML_VK_DISABLE_GRAPH_OPTIMIZE=1 is not a benchmarking workaround either as it does not fix the underlying performance code; it only fixes correctness, potentially at the cost of performance. |
|
I'm merging this PR now as it's already a good baseline. For bug reports, please fire separate issues |
This comment was marked as outdated.
This comment was marked as outdated.
|
hmm bad news then, this PR is created from an org, I cannot push to it 
|
…27342) (#27816) * spec : add DFlash2 support (local convolution + candidate selector) (#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co> * revert top-k.cu changes --------- Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn> Co-authored-by: Jian Chen <jianchen0311@gmail.com>
Upstream merged PR ggml-org#27342 as b10f9ca, so the fork's own port of that PR is now a duplicate. Per the refresh rule, take the upstream implementation and drop the fork hunks: greedy selection with a p_min threshold replaces the fork's stochastic maximal-coupling sampling, and the encoder path gains upstream's M-RoPE position handling. This removes selector_rng, selector_reset and the dists-based DFlash2 sampling. Keep Nathan's capability guard from 0b0f35d on top: a DSV4 backbone must still be refused explicitly, because its graph does not build the selector lattice. Align llama_model_dflash::graph with upstream, which no longer overrides build_post_sampling. conversion/qwen.py and gguf-py/gguf/constants.py take upstream verbatim: an interleaved-rope permute the fork lacked, and pure alignment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
does this fix dflash2 running on multi-gpu setups ? -sm tensor not working on previous iterations |
Nope, still there:
|
Merge upstream PR ggml-org#27342 at 2f3923b into BeeLlama v0.4.4 at 1ce739a (common base 9731ad3). The merge had no textual conflicts. The reviewed integration preserves BeeLlama speculative state restore, DSpark, reduced-vocabulary, KVarN, and KV-tail behavior while adding DFlash2 convolution, selector, M-RoPE, converter/GGUF plumbing, and wide-row CUDA top-k support. Use -FLT_MAX for empty CUDA top-k candidates to avoid a CUDA 13.1 overflow warning. Verified with the Windows CPU build and 100/100 CPU tests; the CUDA 13.1 sm_86 default build script in shared and all-tests configurations; 100/100 CUDA tests; 80 targeted wide-row CUDA top-k cases and CUDA top-k perf runs; targeted checkpoint/state tests; real legacy DFlash conversion and speculative decoding; real DFlash2 conversion, GGUF metadata/tensor inspection, and model tensor validation.
…gml-org#27342) * support DFlash2 * Add p_min in DFlash2 Assisted-by: Claude Opus 5 * Revert unnecessary changes Assisted-by: Claude Opus 5 * Revert draft sampling in rejection sampling Assisted-by: Claude Opus 5 * Refactor code structure Assisted-by: Claude Opus 5 * Delete embedding scaling Assisted-by: Claude Opus 5 * Gate output transforms on DFlash2 Assisted-by: Claude Opus 5 * Optimize Dflash 2 cost Assisted-by: Claude Opus 5 * Avoid using atoi Assisted-by: Claude Opus 5 * Modify comments Assisted-by: Claude Opus 5 * Move llama_model_dflash_selector_top_k to llama-ext.h Assisted-by: Claude Opus 5 * Formatting Assisted-by: Claude Opus 5 * Apply patch to fix the mrope bug Assisted-by: Claude Opus 5 * fix ci Assisted-by: Claude Opus 5 * Fix graph number calculation Assisted-by: Claude Opus 5 * rename hid and unary Assisted-by: Claude Opus 5 --------- Co-authored-by: Jian Chen <jianchen0311@gmail.com> Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
|
@MEsfits @TheFuchsen PR to fix |
|
Important GGUF generated before Aug 27th 2026 must be reconverted, otherwise vision will be broken. See: #27342 (comment) Many broken GGUFs are still being distributed from Please, ask them to reconvert GGUF, or simply download from official |
@ngxson Are these the correct GGUFs: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF? Because using Q4KM of the above repos when i give it an image, the acceptance falls down severely to less than 0.03 Update: incoai just upload the GGUFs, I'm seeing the same issue: |
Upstream's merged DFlash2 (b10f9ca, ggml-org#27342) builds the selector from the decoder graph constructor via build_dflash2_selector, and has no build_post_sampling hook at all. The hook and its call site came from the PR-stage version cherry-picked earlier; the upstream merge removed the call site, so the re-added override was never reached. Removes the duplicate implementation and the now-unused virtual, which puts llama-graph.h back at parity with upstream. Assisted-by: Claude Opus 5
20 participants
Important
GGUF generated before Aug 27th 2026 must be reconverted, otherwise vision will be broken. See: #27342 (comment)
Overview
DFlash2 adds two more modules compared to DFlash, grouped dynamic depthwise convolution and candidate selector.
The formula is for reference.
out[i,c] = Σ_t (base[t,c] + δ[i,t,g(c)]) · x[i−t,c].baseis the static kernel,δis the dynamic part predicted from the input.xis divided into groupsg(c)andδis shared across a group. Another one isedge(p→c) = ⟨A[p] ⊙ project(h), B[c]⟩ + unary[c].A/Bare the codebooks for the predecessor and the candidate. This position's hidden is projected byproject(h)to the same rank to multiplyA. Andunary[c]is the draft's own score forc.DFlash2 is enabled when the checkpoint is DFlash2; no need to use extra flag.
We evaled
Qwen3.8-27BatQ4_K_Mon Apple M5 Pro 64 GB. And the test data is first 8 problems of GSM8K.Concurrency is set to 1 and temperature is 1.0, top-p 0.95, top-k 20. We use default
xhighreasoning and set2,048 maximum new tokens.
Requirements